{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mNcswZ2J61mM"
      },
      "source": [
        "# Building a Hybrid RAG System with PydanticAI and MongoDB: Creating an AI Agent for Tech News Search and Retrieval\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mongodb-developer/GenAI-Showcase/blob/main/notebooks/agents/ai_agent_with_pydanticai_and_mongodb.ipynb)\n",
        "\n",
        "[![Watch on YouTube](https://img.youtube.com/vi/2HPQKIGwQV0/hqdefault.jpg)](https://www.youtube.com/watch/2HPQKIGwQV0?si=Kvlm_VmqWS1J4YTxQ)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bjdyKvcO696b"
      },
      "source": [
        "This project demonstrates how to build an intelligent information retrieval system that combines local vector search with real-time internet search capabilities. The system addresses a critical challenge in AI applications: providing accurate, up-to-date information while maintaining high performance and reliability.\n",
        "\n",
        "Core Problem and Solution:\n",
        "\n",
        "Traditional language models often provide outdated information, while pure internet search lacks semantic understanding. This hybrid RAG system solves this by combining a MongoDB vector database for historical tech news with real-time internet search through Tavily, ensuring both speed and freshness of information.\n",
        "\n",
        "---\n",
        "\n",
        "Real-World Applications:\n",
        "- **Enterprise Intelligence**: Companies like Microsoft or Google could use this to track competitor activities and market trends, combining historical data with breaking news.\n",
        "\n",
        "- **Financial Analysis**: Investment firms could monitor market movements and company developments, getting both archived research and current announcements about topics like \"recent developments in EV technology.\"\n",
        "\n",
        "- **Research Monitoring**: Research institutions could track scientific developments, accessing both established research and recent breakthroughs in fields like quantum computing or AI.\n",
        "\n",
        "---\n",
        "\n",
        "What makes this implementation particularly powerful is its use of modern tools and practices:\n",
        "\n",
        "- Pydantic provides robust type safety and validation\n",
        "- PydanticAI for implementing ai agents with access to system tools\n",
        "- MongoDB's vector search capabilities enable efficient semantic search\n",
        "- Tavily for internet search and implementing HybridRAG\n",
        "- The hybrid RAG approach combines the benefits of both local and internet search\n",
        "- Dependency injection patterns make the system maintainable and testable\n",
        "\n",
        "---\n",
        "\n",
        "Glossary:\n",
        "- Agents\n",
        "- RAG\n",
        "- Agentic RAG\n",
        "- Control Flow\n",
        "- HybridRAG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cL7iN7BQ80eJ"
      },
      "source": [
        "## Step 1: Installing Libaries and Environment Variables\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "TNDMZmNA2VYS"
      },
      "outputs": [],
      "source": [
        "!pip install -U --quiet pydantic-ai pymongo datasets pandas tavily-python"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "69njTg9GAgN3"
      },
      "source": [
        "Let's break down why we need each of these packages:\n",
        "\n",
        "- `pydantic-ai`: This is our framework for building type-safe AI agents. It provides the scaffolding for creating reliable, maintainable AI applications with proper dependency injection and error handling.\n",
        "- `pymongo`: Our interface to MongoDB, which will serve as both our vector database and operational data store. We'll use it to store and retrieve embeddings for semantic search.\n",
        "- `datasets`: Hugging Face's datasets library, which we'll use to load our initial tech news dataset. This gives us a solid foundation of data to work with and use as the knowledge base for the agent.\n",
        "- `pandas`: The Swiss Army knife of data manipulation in Python. We'll use it to process and transform our data before storage.\n",
        "- `tavily-python`: A powerful search client that will enable our system to perform real-time internet searches, complementing our local vector search capabilities.\n",
        "\n",
        "\n",
        "\n",
        "> Note: As of the publshing of this notebook, PydanticAI is in early beta."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "SZJ8Gj9x32Rw"
      },
      "outputs": [],
      "source": [
        "import getpass\n",
        "import os\n",
        "\n",
        "\n",
        "# Function to securely get and set environment variables\n",
        "def set_env_securely(var_name, prompt):\n",
        "    value = getpass.getpass(prompt)\n",
        "    os.environ[var_name] = value"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zol2tBtO34UP",
        "outputId": "ce0977ed-0111-4ddb-943a-0c7e58933a91"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your OpenAI API key: ··········\n"
          ]
        }
      ],
      "source": [
        "# Get your OpenAI Key: https://platform.openai.com/api-keys\n",
        "set_env_securely(\"OPENAI_API_KEY\", \"Enter your OpenAI API key: \")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b0rj2u5k9Yaa"
      },
      "source": [
        "## Step 2: Creating a Simple Agent with PydanticAI and OpenAI\n",
        "\n",
        "\n",
        "The first major component we need to understand is the AI agent itself. This agent will be the orchestrator of our entire system, handling everything from query processing to result generation.\n",
        "\n",
        "\n",
        "*An agent is a computational entity composed of several integrated components, including the brain(llm), perception(environment) and action components(tools). These components work cohesively to enable the agent to achieve its defined objectives and goals.*\n",
        "\n",
        "Read more [here](https://www.mongodb.com/resources/basics/artificial-intelligence/ai-agents)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eY-Gatqk_DQ6"
      },
      "source": [
        "![image.png]()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "5C5eRA3T2f6t"
      },
      "outputs": [],
      "source": [
        "import nest_asyncio\n",
        "from pydantic_ai import Agent\n",
        "from pydantic_ai.models.openai import OpenAIModel\n",
        "\n",
        "# Apply nest_asyncio patch to allow nested event loops\n",
        "nest_asyncio.apply()\n",
        "\n",
        "model = OpenAIModel(\"gpt-4o\")\n",
        "\n",
        "agent = Agent(\n",
        "    model,\n",
        "    system_prompt=\"When provided with a sentence, simulate shouting\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ozj7mtXi-YUl"
      },
      "source": [
        "In modern Python applications, especially those dealing with AI and real-time data processing, we often work with asynchronous operations. However, when working in environments like Jupyter notebooks or when dealing with nested event loops, we can run into limitations with Python's default asyncio implementation.\n",
        "\n",
        "The line `nest_asyncio.apply()` solves this by allowing nested event loops to run. Think of it like giving your code the ability to multitask within multitasking – it's essential for complex applications that need to handle multiple asynchronous operations simultaneously.\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "**Pydantic AI Approach to Agents**\n",
        "\n",
        "When building AI applications, one of the most crucial components is managing interactions with language models in a structured, type-safe way. PydanticAI's Agent system provides exactly this, offering a robust framework for creating AI-powered applications.\n",
        "\n",
        "A high level way of conceptualizing an Agent `Agent()` is to imagine a wrapper around an LLM `model = OpenAIModel('gpt-4o')` that converts the LLM into a system compoents with additional components such as:\n",
        "\n",
        "- System Prompts: Instructions that guide the LLM's behavior\n",
        "- Function Tools: Custom functions the LLM can call during execution\n",
        "- Structured Result Types: Defined output formats\n",
        "- Dependencies: Resources needed during execution\n",
        "- Model Settings: Configuration for fine-tuning responses\n",
        "\n",
        "We will talk more on dependencies and other components in later section of this notebook. But in the code above we pass the `model` and `system_prompt` arguments. These are the only component we need for a basic level Agent using Pydantic.\n",
        "\n",
        "For our basic Agent, it will take a user input and then convert it into uppercase letters to simulate shouting\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L1fGY3Nl8DYz",
        "outputId": "92f22312-028b-4691-e539-a62483d6b8ba"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "THIS SENTENCE STARTED OFF INITIALLY QUIETER!\n"
          ]
        }
      ],
      "source": [
        "result = agent.run_sync(\"this sentence started off initially quieter\")\n",
        "print(result.data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nw9HIrXNFVwM"
      },
      "source": [
        "There are three main ways to execute an agent in Pydantic AI:\n",
        "\n",
        "1. `agent.run()`: This is used in asynchronous contexts where you want to await a single complete response. It returns a coroutine that resolves to a RunResult containing the agent's full response. Ideal for async web applications or when integrating with other async code.\n",
        "\n",
        "2. `agent.run_sync()`: This is used in synchronous contexts where you want a simple, blocking call that returns a complete response. It's essentially a wrapper around run() that handles the async/sync conversion for you. Perfect for scripts, notebooks, or synchronous applications where you need a straightforward way to get results. This is the one we use in the example above.\n",
        "\n",
        "3. `agent.run_stream()`: This is used when you want to receive the agent's response in chunks as they become available. It returns a StreamedRunResult that can be iterated over asynchronously, making it ideal for real-time applications, chat interfaces, or when dealing with long responses where you want to show progressive updates to users."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HU8BqHUNGMyU"
      },
      "source": [
        "And that's how simple it is to build an Agent with Pydantic AI. It's very straightforward yet powerful, offering type safety, dependency injection(shown later), and flexible execution methods all in one package.\n",
        "\n",
        "While the basic setup can be as simple as defining a model and system prompt, the framework scales elegantly to handle complex use cases like our hybrid RAG system.\n",
        "\n",
        "The combination of Python's type system with PydanticAI's structured approach to AI agent development makes it an excellent choice for building production-ready AI applications that are both maintainable and reliable.\n",
        "\n",
        "Next, let's give our Agent some knowledge."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_SZUam4nGt83"
      },
      "source": [
        "## Step 3: Data Loading and Preparation\n",
        "\n",
        "One of the first steps in building a robust RAG system is establishing a solid knowledge base. Let's explore how to efficiently load and process data from Hugging Face's datasets library, specifically focusing on a tech news embeddings dataset.\n",
        "\n",
        "To further enhance the dataset's utility, there is an e,embedding attribute for each data point that has a vector embedding created using the OpenAI EMBEDDING_MODEL = \"text-embedding-3-small\", with an EMBEDDING_DIMENSION of 256.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "32810c0d52ce427aa2b7ac56c7c773b3",
            "df49ab197ef646ab8ef0054c3f7356d4",
            "20ac8b6e85984a7caa9fa25ecee2fefd",
            "54c36c1e4b124d1ea8a1761ec50bad89",
            "f690099db2294a1a8bfe6844fc164c69",
            "90c496cef99d44a2a9158f1ee6582873",
            "2228e72308b04108ba71d63834614646",
            "f6cb7ce3936544bfa36493ba1e52df54",
            "1ed1495a5d184329a2ed7a337f166f0f",
            "ab5f3f777ba246ce9b044e2658109bb3",
            "00a23ef470614479aa83f44599c78c75"
          ]
        },
        "id": "XwJ_a1It5Kr2",
        "outputId": "f23b7f98-a65a-4a21-86c6-616adf6af361"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "32810c0d52ce427aa2b7ac56c7c773b3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Resolving data files:   0%|          | 0/42 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "from datasets import load_dataset\n",
        "\n",
        "# Ensure you have an HF_TOKEN in your environment variable\n",
        "# https://huggingface.co/datasets/MongoDB/tech-news-embeddings\n",
        "ds = load_dataset(\"MongoDB/tech-news-embeddings\", split=\"train\", streaming=True)\n",
        "\n",
        "dataset_segment = ds.take(10000)\n",
        "\n",
        "dataset_df = pd.DataFrame(dataset_segment)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j7-VFbuBHT7q"
      },
      "source": [
        "The code snippet above retrieves data from the Hugging Face dataset repository, specifically, the MongnoDB tech_embeddings dataset available [here](https://huggingface.co/datasets/MongoDB/tech-news-embeddings).\n",
        "\n",
        "\n",
        "- Streaming Mode: By setting `streaming=True`, we enable efficient memory handling for large datasets. Instead of loading everything at once, we can process data in chunks.\n",
        "\n",
        "- Split Selection: Using `split=\"train\"` specifies which portion of the dataset we want to access.\n",
        "\n",
        "- Memory Management: `ds.take(10000)` allows us to work with a subset of data. You can increase this as you see fit. The limitation primarly prevents memory overflow issues with large datasets. The full tech embeddings dataset contains 1,576,528 data points.\n",
        "\n",
        "\n",
        "> Note: best practices for when working with datasets for RAG systems is to start small. Begin with a manageable subset to validate your pipeline and then incrementally increase size once core functionality is verified\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 590
        },
        "id": "P-ik5M2PHSyx",
        "outputId": "ab0b2c59-4a32-4a0d-ad32-79306941bffa"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"dataset_df\",\n  \"rows\": 10000,\n  \"fields\": [\n    {\n      \"column\": \"_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10000,\n        \"samples\": [\n          \"65c64730f187c085a8670eec\",\n          \"65c644daf187c085a86708cc\",\n          \"65c64123f187c085a866fd43\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"companyName\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 11,\n        \"samples\": [\n          \"360 SECURITY\",\n          \"01Synergy\",\n          \"Cloud Spot\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"companyUrl\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 11,\n        \"samples\": [\n          \"https://hackernoon.com/company/360security\",\n          \"https://hackernoon.com/company/01synergy\",\n          \"https://hackernoon.com/company/cloudspot\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"published_at\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 3786,\n        \"samples\": [\n          \"2023-04-04 23:59:00\",\n          \"2023-02-13 13:45:00\",\n          \"2023-08-16 22:58:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"url\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3739,\n        \"samples\": [\n          \"https://www.benzinga.com/pressreleases/23/04/g31954116/sios-technology-announces-cloud-availability-symposium-2023-disaster-recovery-mastery-unveils-line\",\n          \"https://www.tmcnet.com/usubmit/-lumen-taps-syndio-advance-workplace-equity-transparency-/2023/03/08/9773388.htm\",\n          \"https://www.theguardian.com/world/2023/aug/25/how-the-eu-digital-services-act-affects-facebook-google-and-others\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"title\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3410,\n        \"samples\": [\n          \"3-159 off 14 overs: Experimental Australia roll out the popgun pies but down Pakistan\",\n          \"911 call mixup leads crews to wrong home busted door debate on best practices\",\n          \"Find free newcomer services near you\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"main_image\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1300,\n        \"samples\": [\n          \"https://www.bing.com/th?id=OVFT.mJWyAWFg6nzMEstlL3TT1y&pid=News\",\n          \"https://www.bing.com/th?id=OVFT.jZWuO2KMP0TgiD80NYW_US&pid=News\",\n          \"https://www.bing.com/th?id=OVFT.-QHPM4b205vusrTzGy9iyy&pid=News\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"description\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3716,\n        \"samples\": [\n          \"There is a total of 159 vacancies for the post of Assistant Provident Fund Commissioner in the Employee\\u2019s Provident Fund Organization. Candidates can apply online for the APFC Recruitment till 17 March 2023. Here candidates can check the complete details ...\",\n          \"IT IS REPORTED that in a meeting between Civil War generals Grant and Sherman that Grant marveled at the technology they had at their disposal. With the steamboat railroad and the telegraph ...\",\n          \"Amazon and other tech companies are chomping at the bit to use drones for delivery and other services ... According to a Feb. 1 story in The Information the Federal Aviation Administration ...\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "dataset_df"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-b42d5c5b-4a54-42b1-8f69-0a1fb4b7797b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>_id</th>\n",
              "      <th>companyName</th>\n",
              "      <th>companyUrl</th>\n",
              "      <th>published_at</th>\n",
              "      <th>url</th>\n",
              "      <th>title</th>\n",
              "      <th>main_image</th>\n",
              "      <th>description</th>\n",
              "      <th>embedding</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>65c63ea1f187c085a866f680</td>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-16 02:09:00</td>\n",
              "      <td>https://www.businesswire.com/news/home/2023051...</td>\n",
              "      <td>onsemi and Sineng Electric Spearhead the Devel...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>(Nasdaq: ON) a leader in intelligent power and...</td>\n",
              "      <td>[0.05243798345327377, -0.10347484797239304, -0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>65c63ea2f187c085a866f681</td>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-02 00:07:00</td>\n",
              "      <td>https://elkodaily.com/news/local/adobe-student...</td>\n",
              "      <td>Adobe student receives national Information an...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>ELKO — An eighth grader at Adobe Middle School...</td>\n",
              "      <td>[0.0036485784221440554, -0.05992984399199486, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>65c63ea2f187c085a866f682</td>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-01 22:22:00</td>\n",
              "      <td>https://www.aei.org/technology-and-innovation/...</td>\n",
              "      <td>Modernizing State Services: Harnessing Technol...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>To deliver 21st-century government services Go...</td>\n",
              "      <td>[0.012319465167820454, -0.0807630866765976, 0....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>65c63ea2f187c085a866f683</td>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-02 13:12:00</td>\n",
              "      <td>https://www.crn.com/news/managed-services/terr...</td>\n",
              "      <td>Terry Richardson On Why He Left AMD GreenPages...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>In February GreenPages acquired Toronto-based ...</td>\n",
              "      <td>[-0.02363203465938568, 0.021521812304854393, 0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>65c63ea7f187c085a866f684</td>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-15 20:01:00</td>\n",
              "      <td>https://www.benzinga.com/pressreleases/23/05/3...</td>\n",
              "      <td>Synex Renewable Energy Corporation (Formerly S...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>The conference will bring together growth orie...</td>\n",
              "      <td>[0.08473014086484909, -0.07019763439893723, 0....</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b42d5c5b-4a54-42b1-8f69-0a1fb4b7797b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b42d5c5b-4a54-42b1-8f69-0a1fb4b7797b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b42d5c5b-4a54-42b1-8f69-0a1fb4b7797b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6a57f21b-1312-4af6-a269-21d80626978e\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6a57f21b-1312-4af6-a269-21d80626978e')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6a57f21b-1312-4af6-a269-21d80626978e button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                        _id companyName  \\\n",
              "0  65c63ea1f187c085a866f680   01Synergy   \n",
              "1  65c63ea2f187c085a866f681   01Synergy   \n",
              "2  65c63ea2f187c085a866f682   01Synergy   \n",
              "3  65c63ea2f187c085a866f683   01Synergy   \n",
              "4  65c63ea7f187c085a866f684   01Synergy   \n",
              "\n",
              "                                 companyUrl         published_at  \\\n",
              "0  https://hackernoon.com/company/01synergy  2023-05-16 02:09:00   \n",
              "1  https://hackernoon.com/company/01synergy  2023-05-02 00:07:00   \n",
              "2  https://hackernoon.com/company/01synergy  2023-05-01 22:22:00   \n",
              "3  https://hackernoon.com/company/01synergy  2023-05-02 13:12:00   \n",
              "4  https://hackernoon.com/company/01synergy  2023-05-15 20:01:00   \n",
              "\n",
              "                                                 url  \\\n",
              "0  https://www.businesswire.com/news/home/2023051...   \n",
              "1  https://elkodaily.com/news/local/adobe-student...   \n",
              "2  https://www.aei.org/technology-and-innovation/...   \n",
              "3  https://www.crn.com/news/managed-services/terr...   \n",
              "4  https://www.benzinga.com/pressreleases/23/05/3...   \n",
              "\n",
              "                                               title  \\\n",
              "0  onsemi and Sineng Electric Spearhead the Devel...   \n",
              "1  Adobe student receives national Information an...   \n",
              "2  Modernizing State Services: Harnessing Technol...   \n",
              "3  Terry Richardson On Why He Left AMD GreenPages...   \n",
              "4  Synex Renewable Energy Corporation (Formerly S...   \n",
              "\n",
              "                                          main_image  \\\n",
              "0  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "1  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "2  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "3  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "4  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "\n",
              "                                         description  \\\n",
              "0  (Nasdaq: ON) a leader in intelligent power and...   \n",
              "1  ELKO — An eighth grader at Adobe Middle School...   \n",
              "2  To deliver 21st-century government services Go...   \n",
              "3  In February GreenPages acquired Toronto-based ...   \n",
              "4  The conference will bring together growth orie...   \n",
              "\n",
              "                                           embedding  \n",
              "0  [0.05243798345327377, -0.10347484797239304, -0...  \n",
              "1  [0.0036485784221440554, -0.05992984399199486, ...  \n",
              "2  [0.012319465167820454, -0.0807630866765976, 0....  \n",
              "3  [-0.02363203465938568, 0.021521812304854393, 0...  \n",
              "4  [0.08473014086484909, -0.07019763439893723, 0....  "
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Observe top data points\n",
        "dataset_df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 590
        },
        "id": "TX0zw7ni3FDO",
        "outputId": "493cc1d9-4ec8-4573-de80-59baf593060c"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"dataset_df\",\n  \"rows\": 10000,\n  \"fields\": [\n    {\n      \"column\": \"companyName\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 11,\n        \"samples\": [\n          \"360 SECURITY\",\n          \"01Synergy\",\n          \"Cloud Spot\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"companyUrl\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 11,\n        \"samples\": [\n          \"https://hackernoon.com/company/360security\",\n          \"https://hackernoon.com/company/01synergy\",\n          \"https://hackernoon.com/company/cloudspot\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"published_at\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 3786,\n        \"samples\": [\n          \"2023-04-04 23:59:00\",\n          \"2023-02-13 13:45:00\",\n          \"2023-08-16 22:58:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"url\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3739,\n        \"samples\": [\n          \"https://www.benzinga.com/pressreleases/23/04/g31954116/sios-technology-announces-cloud-availability-symposium-2023-disaster-recovery-mastery-unveils-line\",\n          \"https://www.tmcnet.com/usubmit/-lumen-taps-syndio-advance-workplace-equity-transparency-/2023/03/08/9773388.htm\",\n          \"https://www.theguardian.com/world/2023/aug/25/how-the-eu-digital-services-act-affects-facebook-google-and-others\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"title\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3410,\n        \"samples\": [\n          \"3-159 off 14 overs: Experimental Australia roll out the popgun pies but down Pakistan\",\n          \"911 call mixup leads crews to wrong home busted door debate on best practices\",\n          \"Find free newcomer services near you\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"main_image\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1300,\n        \"samples\": [\n          \"https://www.bing.com/th?id=OVFT.mJWyAWFg6nzMEstlL3TT1y&pid=News\",\n          \"https://www.bing.com/th?id=OVFT.jZWuO2KMP0TgiD80NYW_US&pid=News\",\n          \"https://www.bing.com/th?id=OVFT.-QHPM4b205vusrTzGy9iyy&pid=News\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"description\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3716,\n        \"samples\": [\n          \"There is a total of 159 vacancies for the post of Assistant Provident Fund Commissioner in the Employee\\u2019s Provident Fund Organization. Candidates can apply online for the APFC Recruitment till 17 March 2023. Here candidates can check the complete details ...\",\n          \"IT IS REPORTED that in a meeting between Civil War generals Grant and Sherman that Grant marveled at the technology they had at their disposal. With the steamboat railroad and the telegraph ...\",\n          \"Amazon and other tech companies are chomping at the bit to use drones for delivery and other services ... According to a Feb. 1 story in The Information the Federal Aviation Administration ...\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "dataset_df"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-a518754c-c526-4c56-b1f7-3a32f983fb08\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>companyName</th>\n",
              "      <th>companyUrl</th>\n",
              "      <th>published_at</th>\n",
              "      <th>url</th>\n",
              "      <th>title</th>\n",
              "      <th>main_image</th>\n",
              "      <th>description</th>\n",
              "      <th>embedding</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-16 02:09:00</td>\n",
              "      <td>https://www.businesswire.com/news/home/2023051...</td>\n",
              "      <td>onsemi and Sineng Electric Spearhead the Devel...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>(Nasdaq: ON) a leader in intelligent power and...</td>\n",
              "      <td>[0.05243798345327377, -0.10347484797239304, -0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-02 00:07:00</td>\n",
              "      <td>https://elkodaily.com/news/local/adobe-student...</td>\n",
              "      <td>Adobe student receives national Information an...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>ELKO — An eighth grader at Adobe Middle School...</td>\n",
              "      <td>[0.0036485784221440554, -0.05992984399199486, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-01 22:22:00</td>\n",
              "      <td>https://www.aei.org/technology-and-innovation/...</td>\n",
              "      <td>Modernizing State Services: Harnessing Technol...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>To deliver 21st-century government services Go...</td>\n",
              "      <td>[0.012319465167820454, -0.0807630866765976, 0....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-02 13:12:00</td>\n",
              "      <td>https://www.crn.com/news/managed-services/terr...</td>\n",
              "      <td>Terry Richardson On Why He Left AMD GreenPages...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>In February GreenPages acquired Toronto-based ...</td>\n",
              "      <td>[-0.02363203465938568, 0.021521812304854393, 0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>01Synergy</td>\n",
              "      <td>https://hackernoon.com/company/01synergy</td>\n",
              "      <td>2023-05-15 20:01:00</td>\n",
              "      <td>https://www.benzinga.com/pressreleases/23/05/3...</td>\n",
              "      <td>Synex Renewable Energy Corporation (Formerly S...</td>\n",
              "      <td>https://firebasestorage.googleapis.com/v0/b/ha...</td>\n",
              "      <td>The conference will bring together growth orie...</td>\n",
              "      <td>[0.08473014086484909, -0.07019763439893723, 0....</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a518754c-c526-4c56-b1f7-3a32f983fb08')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a518754c-c526-4c56-b1f7-3a32f983fb08 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a518754c-c526-4c56-b1f7-3a32f983fb08');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-253ed125-7308-4732-bd41-05f9cfbfd888\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-253ed125-7308-4732-bd41-05f9cfbfd888')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-253ed125-7308-4732-bd41-05f9cfbfd888 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "  companyName                                companyUrl         published_at  \\\n",
              "0   01Synergy  https://hackernoon.com/company/01synergy  2023-05-16 02:09:00   \n",
              "1   01Synergy  https://hackernoon.com/company/01synergy  2023-05-02 00:07:00   \n",
              "2   01Synergy  https://hackernoon.com/company/01synergy  2023-05-01 22:22:00   \n",
              "3   01Synergy  https://hackernoon.com/company/01synergy  2023-05-02 13:12:00   \n",
              "4   01Synergy  https://hackernoon.com/company/01synergy  2023-05-15 20:01:00   \n",
              "\n",
              "                                                 url  \\\n",
              "0  https://www.businesswire.com/news/home/2023051...   \n",
              "1  https://elkodaily.com/news/local/adobe-student...   \n",
              "2  https://www.aei.org/technology-and-innovation/...   \n",
              "3  https://www.crn.com/news/managed-services/terr...   \n",
              "4  https://www.benzinga.com/pressreleases/23/05/3...   \n",
              "\n",
              "                                               title  \\\n",
              "0  onsemi and Sineng Electric Spearhead the Devel...   \n",
              "1  Adobe student receives national Information an...   \n",
              "2  Modernizing State Services: Harnessing Technol...   \n",
              "3  Terry Richardson On Why He Left AMD GreenPages...   \n",
              "4  Synex Renewable Energy Corporation (Formerly S...   \n",
              "\n",
              "                                          main_image  \\\n",
              "0  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "1  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "2  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "3  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "4  https://firebasestorage.googleapis.com/v0/b/ha...   \n",
              "\n",
              "                                         description  \\\n",
              "0  (Nasdaq: ON) a leader in intelligent power and...   \n",
              "1  ELKO — An eighth grader at Adobe Middle School...   \n",
              "2  To deliver 21st-century government services Go...   \n",
              "3  In February GreenPages acquired Toronto-based ...   \n",
              "4  The conference will bring together growth orie...   \n",
              "\n",
              "                                           embedding  \n",
              "0  [0.05243798345327377, -0.10347484797239304, -0...  \n",
              "1  [0.0036485784221440554, -0.05992984399199486, ...  \n",
              "2  [0.012319465167820454, -0.0807630866765976, 0....  \n",
              "3  [-0.02363203465938568, 0.021521812304854393, 0...  \n",
              "4  [0.08473014086484909, -0.07019763439893723, 0....  "
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Remove the _id field\n",
        "dataset_df = dataset_df.drop(columns=[\"_id\"])\n",
        "\n",
        "# Observe top data points\n",
        "dataset_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2TaTtXlBIhQt"
      },
      "source": [
        "PydanticAI is created by the team that made Pydantic, which we will use to ensure data integrity and type safety throughout our application.\n",
        "\n",
        "Below we create a data model `TechNewsData`. This structured approach to data modeling helps prevent bugs early in the development process and makes your code more maintainable. As your RAG system grows, having these strong type guarantees becomes increasingly valuable.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "Oro4sxzRnoKn"
      },
      "outputs": [],
      "source": [
        "from typing import List, Optional\n",
        "\n",
        "from pydantic import BaseModel, Field\n",
        "\n",
        "\n",
        "class TechNewsData(BaseModel):\n",
        "    companyName: str\n",
        "    companyUrl: str\n",
        "    published_at: str\n",
        "    title: str\n",
        "    description: str\n",
        "    url: str\n",
        "    embedding: List[float] = Field(\n",
        "        ..., description=\"The embedding vector for the news article\"\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lmY4b6wtJHRa"
      },
      "source": [
        "The benefits of using Pydantic models in your RAG system are threefold:\n",
        "\n",
        "1. you get robust type safety with automatic validation, clear contracts, and IDE autocompletion; serialization capabilities that make JSON conversion and MongoDB integration effortless while maintaining clean API interfaces;\n",
        "\n",
        "2. and comprehensive documentation features including self-documenting code, clear field descriptions,\n",
        "\n",
        "3. and automatic schema generation, all of which contribute to making your codebase more maintainable and developer-friendly.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "kp9D-yv0rB6_"
      },
      "outputs": [],
      "source": [
        "# Conform every datapoint to the TechNewsData model\n",
        "dataset_df = dataset_df.apply(\n",
        "    lambda x: TechNewsData(**x.to_dict()).model_dump(), axis=1\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SP5gKb6DJ-cx"
      },
      "source": [
        "## Step 4: Defining Embedding Function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XhHLM9q-ZUMF"
      },
      "source": [
        "When building a RAG (Retrieval Augmented Generation) system, one of the fundamental components is converting text into vector embeddings. These embeddings allow us to perform semantic search, finding similar content based on meaning rather than just matching keywords. Let's look at how to implement this using OpenAI's embedding API.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "91vOmQQd7VJG"
      },
      "outputs": [],
      "source": [
        "import openai\n",
        "\n",
        "client = openai.OpenAI()\n",
        "DIMENSION_SIZE = 256\n",
        "\n",
        "\n",
        "def get_embeddings(\n",
        "    texts: list[str], doc_type: str = \"search_query\"\n",
        ") -> list[list[float]]:\n",
        "    \"\"\"\n",
        "    Generate embeddings for a list of input texts.\n",
        "\n",
        "    Args:\n",
        "        texts: List of strings to generate embeddings for\n",
        "        doc_type: Type of document being embedded (default: \"search_query\")\n",
        "\n",
        "    Returns:\n",
        "        List of embeddings, where each embedding is a list of floats\n",
        "    \"\"\"\n",
        "    # Create embeddings for all texts in a single API call\n",
        "    response = client.embeddings.create(\n",
        "        input=texts, model=\"text-embedding-3-small\", dimensions=DIMENSION_SIZE\n",
        "    )\n",
        "\n",
        "    # Extract embeddings from response\n",
        "    embeddings = [data.embedding for data in response.data]\n",
        "\n",
        "    return embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qpxr0tzPZa32"
      },
      "source": [
        "Embeddings are generated from a list of string input as shown in the example below:\n",
        "```\n",
        "# Example usage in a real-world scenario\n",
        "news_articles = [\n",
        "    \"OpenAI releases GPT-5\",\n",
        "    \"New advancements in quantum computing\",\n",
        "    \"Latest developments in AI ethics\"\n",
        "]\n",
        "\n",
        "```\n",
        "\n",
        "The embedding model used is `text-embedding-3-small`, OpenAI's latest embedding model optimized for efficient semantic search and text similarity tasks. This model offers a good balance between performance and cost, making it suitable for RAG applications.\n",
        "\n",
        "By setting a constant `DIMENSION_SIZE`, we ensure all our embeddings have consistent dimensions. This is crucial when working with vector databases and performing similarity searches. Consistent dimensions are essential because:\n",
        "\n",
        "They enable efficient vector operations\n",
        "They ensure compatibility across your database\n",
        "They allow for predictable memory usage and indexing\n",
        "\n",
        "The dimension size for this notebook is `256`, which is relatively small, and for production scenarios, larger dimension sizes can be used to capture more semantics within the data.\n",
        "\n",
        "When choosing dimension size, consider:\n",
        "\n",
        "- Your specific use case requirements\n",
        "- Available computational resources\n",
        "- Storage capacity\n",
        "- Query performance needs\n",
        "- Cost considerations\n",
        "\n",
        "For many applications, 256 dimensions provide a good starting point for prototyping and testing your RAG system before scaling up to larger dimensions in production.\n",
        "\n",
        "Read these articles for more information on [choosing embedding models](https://www.mongodb.com/developer/products/atlas/choose-embedding-model-rag/) and [chunking stratgeies](https://www.mongodb.com/developer/products/atlas/choosing-chunking-strategy-rag/)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ow0fcgs67_0C"
      },
      "source": [
        "## Step 5: MongoDB (Operational and Vector Database)\n",
        "\n",
        "MongoDB acts as both an operational and vector database for the RAG system.\n",
        "MongoDB Atlas specifically provides a database solution that efficiently stores, queries and retrieves vector embeddings.\n",
        "\n",
        "Creating a database and collection within MongoDB is made simple with MongoDB Atlas.\n",
        "\n",
        "1. First, register for a [MongoDB Atlas account](https://www.mongodb.com/cloud/atlas/register). For existing users, sign into MongoDB Atlas.\n",
        "2. [Follow the instructions](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster/). Select Atlas UI as the procedure to deploy your first cluster.\n",
        "\n",
        "Follow MongoDB’s [steps to get the connection](https://www.mongodb.com/docs/manual/reference/connection-string/) string from the Atlas UI. After setting up the database and obtaining the Atlas cluster connection URI, securely store the URI within your development environment.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "elQtX2oJ8AKM",
        "outputId": "629b8311-3c27-48e7-80ea-ba197e26c109"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your MONGO URI: ··········\n"
          ]
        }
      ],
      "source": [
        "# Set MongoDB URI\n",
        "set_env_securely(\"MONGO_URI\", \"Enter your MONGO URI: \")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "ZuLbyLvg8CHL"
      },
      "outputs": [],
      "source": [
        "import pymongo\n",
        "\n",
        "\n",
        "def get_mongo_client(mongo_uri):\n",
        "    \"\"\"Establish and validate connection to the MongoDB.\"\"\"\n",
        "\n",
        "    client = pymongo.MongoClient(\n",
        "        mongo_uri, appname=\"devrel.showcase.agents.pydanticai.python\"\n",
        "    )\n",
        "\n",
        "    # Validate the connection\n",
        "    ping_result = client.admin.command(\"ping\")\n",
        "    if ping_result.get(\"ok\") == 1.0:\n",
        "        # Connection successful\n",
        "        print(\"Connection to MongoDB successful\")\n",
        "        return client\n",
        "    else:\n",
        "        print(\"Connection to MongoDB failed\")\n",
        "    return None\n",
        "\n",
        "\n",
        "MONGO_URI = os.environ[\"MONGO_URI\"]\n",
        "if not MONGO_URI:\n",
        "    print(\"MONGO_URI not set in environment variables\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WdGF6GAPa4K5"
      },
      "source": [
        "Let's explore how to properly set up and manage MongoDB collections for our RAG system's knowledge base. This setup is crucial for storing and retrieving our vectorized news articles description+title efficiently.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "czifFs9Y8D41",
        "outputId": "0ad2ba73-9466-4150-ea4c-7f0a8496e65e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Connection to MongoDB successful\n",
            "Collection 'knowledge_base' created successfully.\n"
          ]
        }
      ],
      "source": [
        "from pymongo.errors import CollectionInvalid\n",
        "\n",
        "mongo_client = get_mongo_client(MONGO_URI)\n",
        "\n",
        "DB_NAME = \"tech_news_agent\"\n",
        "COLLECTION_NAME = \"knowledge_base\"\n",
        "\n",
        "# Create or get the database\n",
        "db = mongo_client[DB_NAME]\n",
        "\n",
        "# Check if the collection exists\n",
        "if COLLECTION_NAME not in db.list_collection_names():\n",
        "    try:\n",
        "        # Create the collection\n",
        "        db.create_collection(COLLECTION_NAME)\n",
        "        print(f\"Collection '{COLLECTION_NAME}' created successfully.\")\n",
        "    except CollectionInvalid as e:\n",
        "        print(f\"Error creating collection: {e}\")\n",
        "else:\n",
        "    print(f\"Collection '{COLLECTION_NAME}' already exists.\")\n",
        "\n",
        "# Assign the collection\n",
        "knowledge_base = db[COLLECTION_NAME]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EpvB3qiabcFt"
      },
      "source": [
        "When building a RAG system, proper database setup is crucial for managing your knowledge base effectively.\n",
        "\n",
        "Our implementation uses MongoDB, a general purpose database that's particularly well-suited for handling document-based data and vector embeddings.\n",
        "\n",
        "The code snippet above establishes a connection to MongoDB through the `get_mongo_client` function, then sets up a database named \"tech_news_agent\" with a collection called \"knowledge_base\".\n",
        "\n",
        "We implement a robust error-handling pattern that checks if the collection exists before attempting to create it, catching any `CollectionInvalid` exceptions that might occur during the process. This idempotent approach means the code can be run multiple times safely – if the collection already exists, it simply connects to it; if not, it creates it.\n",
        "\n",
        "The clear naming conventions (like `tech_news_agent` for the database and `knowledge_base` for the collection) make the code's purpose immediately apparent and easier to maintain. Finally, we assign the collection to a `knowledge_base` variable, which we'll use throughout our application for storing and retrieving vectorized news articles. This foundation ensures our RAG system has a reliable and efficient data storage layer, ready for implementing vector search capabilities and managing our embedded documents."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O_O_7v5f8HWR"
      },
      "source": [
        "## Step 6: Data Ingestion"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oN4M3Z2p8JPo",
        "outputId": "d5db3dfb-9ea9-4d92-affb-0c19c70ff955"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DeleteResult({'n': 0, 'electionId': ObjectId('7fffffff000000000000003a'), 'opTime': {'ts': Timestamp(1736447697, 1), 't': 58}, 'ok': 1.0, '$clusterTime': {'clusterTime': Timestamp(1736447697, 1), 'signature': {'hash': b'\\x06\\xb1\\xdb^\\xd2cP\\xf5xs\\xba\\xc42x\\x91\\xf1\\x862\\x9bM', 'keyId': 7421923411288391683}}, 'operationTime': Timestamp(1736447697, 1)}, acknowledged=True)"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "knowledge_base.delete_many({})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aX7BUG1Uc0mX"
      },
      "source": [
        "**Why MongoDB for AI Workloads?**\n",
        "\n",
        "MongoDB, offers several compelling advantages for AI workloads, particularly in simplifying data ingestion.\n",
        "\n",
        "The code snippet below demonstrates how MongoDB streamlines the data ingestion process by eliminating the need for explicit serialization and deserialization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "Z-j48WBRvLeG"
      },
      "outputs": [],
      "source": [
        "documents = [TechNewsData(**x).model_dump() for x in dataset_df]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vV-lD-tLuKOm",
        "outputId": "9650f968-0dd9-44b3-e228-55a959f8c463"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Data ingestion into MongoDB completed\n"
          ]
        }
      ],
      "source": [
        "knowledge_base.insert_many(documents)\n",
        "print(\"Data ingestion into MongoDB completed\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rMoOCFjb8SpF"
      },
      "source": [
        "##Step 7: Vector Search Index Creation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "aFlnL2dk8Ow4"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "from pymongo.operations import SearchIndexModel\n",
        "\n",
        "\n",
        "def setup_vector_search_index(collection, index_definition, index_name=\"vector_index\"):\n",
        "    \"\"\"\n",
        "    Setup a vector search index for a MongoDB collection and wait for 30 seconds.\n",
        "\n",
        "    Args:\n",
        "    collection: MongoDB collection object\n",
        "    index_definition: Dictionary containing the index definition\n",
        "    index_name: Name of the index (default: \"vector_index\")\n",
        "    \"\"\"\n",
        "    new_vector_search_index_model = SearchIndexModel(\n",
        "        definition=index_definition, name=index_name, type=\"vectorSearch\"\n",
        "    )\n",
        "\n",
        "    # Create the new index\n",
        "    try:\n",
        "        result = collection.create_search_index(model=new_vector_search_index_model)\n",
        "        print(f\"Creating index '{index_name}'...\")\n",
        "\n",
        "        # Sleep for 60 seconds\n",
        "        print(f\"Waiting for 60 seconds to allow index '{index_name}' to be created...\")\n",
        "        time.sleep(30)\n",
        "\n",
        "        print(f\"60-second wait completed for index '{index_name}'.\")\n",
        "        return result\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error creating new vector search index '{index_name}': {e!s}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "7UnYC7uY8X5S"
      },
      "outputs": [],
      "source": [
        "def create_vector_index_definition():\n",
        "    # Define the field types\n",
        "    base_fields = [\n",
        "        {\n",
        "            \"type\": \"vector\",\n",
        "            \"path\": \"embedding\",  # Ensure this matches the name of the attribute containing vector embeddings in your dataset\n",
        "            \"numDimensions\": DIMENSION_SIZE,\n",
        "            \"similarity\": \"cosine\",\n",
        "        }\n",
        "    ]\n",
        "\n",
        "    return {\"fields\": base_fields}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "tNnWNHJA8o0E"
      },
      "outputs": [],
      "source": [
        "vector_index_definition = create_vector_index_definition()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TEJw_ZeH8p-s",
        "outputId": "37aefefc-1056-49ba-cbdc-d9180f45f58c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'fields': [{'type': 'vector', 'path': 'embedding', 'numDimensions': 256, 'similarity': 'cosine'}]}\n"
          ]
        }
      ],
      "source": [
        "print(vector_index_definition)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "id": "gVpJKI-T8rAr",
        "outputId": "82e15876-16fa-47eb-a247-b2f95606ee8d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Creating index 'vector_index'...\n",
            "Waiting for 60 seconds to allow index 'vector_index' to be created...\n",
            "60-second wait completed for index 'vector_index'.\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'vector_index'"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "setup_vector_search_index(knowledge_base, vector_index_definition, \"vector_index\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0PLkkBIV8th4"
      },
      "source": [
        "## Step 8: Vector Search Operation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "YLgE2id-8s3G"
      },
      "outputs": [],
      "source": [
        "def custom_vector_search(\n",
        "    user_query: list[str],\n",
        "    collection,\n",
        "    embedding_path=\"embedding\",\n",
        "    vector_search_index_name=\"vector_index\",\n",
        "):\n",
        "    \"\"\"\n",
        "    Perform a vector search in the MongoDB collection based on the user query.\n",
        "\n",
        "    Args:\n",
        "        user_query (str): The user's query string.\n",
        "        collection (MongoCollection): The MongoDB collection to search.\n",
        "        embedding_path (str): The path of the embedding field in the documents.\n",
        "        vector_search_index_name (str): The name of the vector search index.\n",
        "\n",
        "    Returns:\n",
        "        list: A list of matching documents.\n",
        "    \"\"\"\n",
        "\n",
        "    # Generate embedding for the user query\n",
        "    query_embedding = get_embeddings(user_query)[0]\n",
        "\n",
        "    if query_embedding is None:\n",
        "        return \"Invalid query or embedding generation failed.\"\n",
        "\n",
        "    # Define the vector search stage\n",
        "    vector_search_stage = {\n",
        "        \"$vectorSearch\": {\n",
        "            \"index\": vector_search_index_name,  # Specifies the index to use for the search\n",
        "            \"queryVector\": query_embedding,  # The vector representing the query\n",
        "            \"path\": embedding_path,  # Field in the documents containing the vectors to search against\n",
        "            \"numCandidates\": 20,  # Number of candidate matches to consider\n",
        "            \"limit\": 5,  # Return top 5 matches\n",
        "        }\n",
        "    }\n",
        "\n",
        "    project_stage = {\n",
        "        \"$project\": {\n",
        "            \"_id\": 0,  # Exclude the _id field\n",
        "            \"title\": 1,  # Include the title field,\n",
        "            \"companyName\": 1,  # Include the companyName field,\n",
        "            \"companyUrl\": 1,  # Include the companyUrl field,\n",
        "            \"published_at\": 1,  # Include the published_at field,\n",
        "            \"description\": 1,  # Include the description field\n",
        "            \"url\": 1,  # Include the url field\n",
        "            \"score\": {\n",
        "                \"$meta\": \"vectorSearchScore\"  # Include the search score\n",
        "            },\n",
        "        }\n",
        "    }\n",
        "\n",
        "    # Define the aggregate pipeline with the vector search stage and additional stages\n",
        "    pipeline = [vector_search_stage, project_stage]\n",
        "\n",
        "    # Execute the explain command\n",
        "    explain_result = collection.database.command(\n",
        "        \"explain\",\n",
        "        {\"aggregate\": collection.name, \"pipeline\": pipeline, \"cursor\": {}},\n",
        "        verbosity=\"executionStats\",\n",
        "    )\n",
        "\n",
        "    # Extract the execution time\n",
        "    vector_search_explain = explain_result[\"stages\"][0][\"$vectorSearch\"]\n",
        "    execution_time_ms = vector_search_explain[\"explain\"][\"query\"][\"stats\"][\"context\"][\n",
        "        \"millisElapsed\"\n",
        "    ]\n",
        "\n",
        "    # Execute the actual query\n",
        "    results = list(collection.aggregate(pipeline))\n",
        "\n",
        "    return results, execution_time_ms"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "tKVHSEXjCxbm"
      },
      "outputs": [],
      "source": [
        "results, time = custom_vector_search(\n",
        "    [\"Get me some news on electric cars if possible\"], knowledge_base\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kDVC1q5mC-t_",
        "outputId": "3156d37e-df72-4f87-aa3e-ca1bbe02591a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{'companyName': '10Clouds',\n",
            "  'companyUrl': 'https://hackernoon.com/company/10clouds',\n",
            "  'description': 'Find insight on Aerojet Rocketdyne Appen and more in the '\n",
            "                 'latest Market Talks covering Technology Media and Telecom.',\n",
            "  'published_at': '2023-03-17 10:52:00',\n",
            "  'score': 0.7128037214279175,\n",
            "  'title': 'Tech Media & Telecom Roundup: Market Talk',\n",
            "  'url': 'https://www.wsj.com/articles/tech-media-telecom-roundup-market-talk-8105659d'},\n",
            " {'companyName': '10Clouds',\n",
            "  'companyUrl': 'https://hackernoon.com/company/10clouds',\n",
            "  'description': 'Find insight on LONGi Green Energy Technology Auto Trader '\n",
            "                 'and more in the latest Market Talks covering the Technology '\n",
            "                 'Media and Telecom sector.',\n",
            "  'published_at': '2023-06-01 11:07:00',\n",
            "  'score': 0.7083801031112671,\n",
            "  'title': 'Tech Media & Telecom Roundup: Market Talk',\n",
            "  'url': 'https://www.wsj.com/articles/tech-media-telecom-roundup-market-talk-8306f871'},\n",
            " {'companyName': '10Clouds',\n",
            "  'companyUrl': 'https://hackernoon.com/company/10clouds',\n",
            "  'description': 'Find insight on Inari Amertron Xiaomi and more in the latest '\n",
            "                 'Market Talks covering the Tech Media & Telecom sector.',\n",
            "  'published_at': '2023-09-01 19:52:00',\n",
            "  'score': 0.7066687941551208,\n",
            "  'title': 'Tech Media & Telecom Roundup: Market Talk',\n",
            "  'url': 'https://www.wsj.com/business/earnings/tech-media-telecom-roundup-market-talk-b499c47a'},\n",
            " {'companyName': '10Clouds',\n",
            "  'companyUrl': 'https://hackernoon.com/company/10clouds',\n",
            "  'description': 'Find insight on Inari Amertron Xiaomi and more in the latest '\n",
            "                 'Market Talks covering the Tech Media & Telecom sector.',\n",
            "  'published_at': '2023-09-01 19:52:00',\n",
            "  'score': 0.7066687941551208,\n",
            "  'title': 'Tech Media & Telecom Roundup: Market Talk',\n",
            "  'url': 'https://www.wsj.com/business/earnings/tech-media-telecom-roundup-market-talk-b499c47a'},\n",
            " {'companyName': '10Clouds',\n",
            "  'companyUrl': 'https://hackernoon.com/company/10clouds',\n",
            "  'description': 'Find insight on Inari Amertron Xiaomi and more in the latest '\n",
            "                 'Market Talks covering the Tech Media & Telecom sector.',\n",
            "  'published_at': '2023-09-01 09:48:00',\n",
            "  'score': 0.7066071033477783,\n",
            "  'title': 'Tech Media & Telecom Roundup: Market Talk',\n",
            "  'url': 'https://www.wsj.com/business/earnings/tech-media-telecom-roundup-market-talk-b499c47a'}]\n"
          ]
        }
      ],
      "source": [
        "import pprint\n",
        "\n",
        "pprint.pprint(results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DgSSSnLmdiif"
      },
      "source": [
        "## Step 9: Creating PydanticAI Agents with Tools"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OSLcK6hs_LdD"
      },
      "source": [
        "![image.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B2ShXemwpuQ8"
      },
      "source": [
        "Tools are a mechanim to extend an agents capabilities in ways that supersedes the limitation of instructions/information provided through system prompts.\n",
        "\n",
        "**Below are the key features of tools in PydanticAI**\n",
        "\n",
        "- Context Awareness: Tools can be either context-aware `(@agent.tool)` or context-free `(@agent.tool_plain)`\n",
        "- Type Safety: Tools leverage Python's type hints for parameter validation\n",
        "- Automatic Documentation: Function docstrings are automatically used to build tool schemas\n",
        "- Flexible Registration: Tools can be registered via decorators or through the Agent constructor"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mHxOpX9cqki3"
      },
      "source": [
        "There are three primary methods to create tools in PydanticAI:\n",
        "\n",
        "1. Using `@agent.tool` Decorator: Tools that need access to dependencies or context (like database connections, API clients). Ideal for most production scenarios where you need to manage resources or maintain state.\n",
        "\n",
        "2. Using `@agent.tool_plain` Decorator:\n",
        "Best for: Stateless utilities or simple computations that don't require context or dependencies. Perfect for pure functions like calculations or text processing.\n",
        "\n",
        "3. Via the tools Parameter in Agent Constructor: Reusing existing functions as tools across different agents or when you need more control over tool configuration. Useful in scenarios where you're building multiple agents that share common functionality.\n",
        "\n",
        "\n",
        "Each method trades off between simplicity and flexibility:\n",
        "\n",
        "- Decorators offer the cleanest syntax and are most commonly used across other similar Agentic Frameworks\n",
        "- Plain tools are perfect for simple, context-free operations\n",
        "- Constructor injection provides the most flexibility for tool reuse and configuration\n",
        "\n",
        "Choose based on your specific needs - whether you need context access, plan to reuse the tool, or prefer a certain style of code organization.\n",
        "\n",
        "We will be implementing all three variaties in the section below\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kXJlJR_l9DGF",
        "outputId": "3e16ac5c-3d0f-4e05-a00c-c0a632d513dc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Connection to MongoDB successful\n"
          ]
        }
      ],
      "source": [
        "from dataclasses import dataclass\n",
        "\n",
        "from pydantic_ai import Agent, RunContext, Tool\n",
        "\n",
        "\n",
        "@dataclass\n",
        "class MongoDeps:\n",
        "    mongo_client = get_mongo_client(MONGO_URI)\n",
        "    db = mongo_client[DB_NAME]\n",
        "    knowledge_base = db[COLLECTION_NAME]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "Fn385-P93KaG"
      },
      "outputs": [],
      "source": [
        "def retrieve_information_from_knowledge_base(\n",
        "    ctx: RunContext[MongoDeps], user_query\n",
        ") -> str:\n",
        "    \"\"\"\n",
        "    Retrieves relevant information from the knowledge base based on the user's query.\n",
        "    Performs a vector search using the provided `user_query` against the `knowledge_base` collection.\n",
        "    \"\"\"\n",
        "    results, execution_time_ms = custom_vector_search(\n",
        "        [user_query], ctx.deps.knowledge_base\n",
        "    )\n",
        "    return str(results)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "lF7O4q0ck11Q"
      },
      "outputs": [],
      "source": [
        "toolbox = [Tool(retrieve_information_from_knowledge_base, takes_ctx=True)]\n",
        "\n",
        "# The option below is also a viable tool implementation, without ability to specify context utilization or not\n",
        "# toolbox = [retrieve_information_from_knowledge_base]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "inoC_YmX3IFm"
      },
      "outputs": [],
      "source": [
        "agent = Agent(\n",
        "    model,\n",
        "    system_prompt=(\"You get the latest news based on a user query\"),\n",
        "    deps_type=MongoDeps,\n",
        "    tools=toolbox,\n",
        "    result_retries=3,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D9CaPN4-C9eF",
        "outputId": "8ac12b66-1b3b-40b9-9d44-30f9310ee8dd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Here are some recent news articles on electric cars:\n",
            "\n",
            "1. **SK signet Inks Deal with Francis Energy for the Supply of Ultra-Fast EV Chargers to the US**  \n",
            "   SK Signet has signed a deal with Francis Energy for an order of more than 1000 EV chargers. Francis Energy is currently the fourth-largest fast charger operator in the United States.  \n",
            "   [Read more](https://www.econotimes.com/SK-signet-Inks-Deal-with-Francis-Energy-for-the-Supply-of-Ultra-Fast-EV-Chargers-to-the-US-1659601) (Published on 2023-07-18)\n",
            "\n",
            "2. **YS Tech working closely with China car vendors**  \n",
            "   Automotive cooling fan supplier Yen Sun Technology (YS Tech) is working closely with Chinese customers and anticipates a new Chinese government policy to boost the country's EV sector.  \n",
            "   [Read more](https://www.digitimes.com/news/a20230309PD211/automotive-china-ev+green-energy-ys-tech.html) (Published on 2023-03-10)\n",
            "\n",
            "3. **Investing in Cleaner Technology: Lesser-Known Areas of Innovation to Watch**  \n",
            "   This article discusses cleaner energy investment opportunities in lesser-known areas of innovation that aim to bridge the gap between current technology and future needs for cleaner energy.  \n",
            "   [Read more](https://www.nasdaq.com/articles/investing-in-cleaner-technology%3A-lesser-known-areas-of-innovation-to-watch) (Published on 2023-01-30)\n",
            "\n",
            "These articles highlight some key developments and investments in the electric car sector and cleaner technologies.\n"
          ]
        }
      ],
      "source": [
        "results = agent.run_sync(\n",
        "    \"Get me some news on electric cars if possible\", deps=MongoDeps\n",
        ")\n",
        "print(results.data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3-Yf6P4mFsYC",
        "outputId": "50489d39-d324-4484-bc0b-995f87de3a67"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your Tavily API key: ··········\n"
          ]
        }
      ],
      "source": [
        "# You can get a Tavily API Key here: https://app.tavily.com/home\n",
        "set_env_securely(\"TAVILY_API_KEY\", \"Enter your Tavily API key: \")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qa8n64QKF4S6",
        "outputId": "40e73b54-d583-4422-ab91-249755c2e48d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collection 'working_memory' created successfully.\n"
          ]
        }
      ],
      "source": [
        "# Create the collection if it doesn't exist\n",
        "if \"working_memory\" not in db.list_collection_names():\n",
        "    working_memory = db.create_collection(\"working_memory\")\n",
        "    print(\"Collection 'working_memory' created successfully.\")\n",
        "else:\n",
        "    working_memory = db[\"working_memory\"]\n",
        "    print(\"Collection 'working_memory' already exists.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "OLEjlWFwIMY-"
      },
      "outputs": [],
      "source": [
        "working_memory_vector_index_definition = create_vector_index_definition()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GCitPV5pGFGA",
        "outputId": "a70439fa-e0a6-448e-b320-4e1e6990b5ca"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Creating index 'vector_index'...\n",
            "Waiting for 60 seconds to allow index 'vector_index' to be created...\n",
            "Error creating new vector search index 'vector_index': 'float' object has no attribute 'sleep'\n"
          ]
        }
      ],
      "source": [
        "setup_vector_search_index(\n",
        "    working_memory, working_memory_vector_index_definition, \"vector_index\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Pxm1FmGrkBdE",
        "outputId": "9c7ebba0-ad01-45b6-f14d-e837bc2ae9c7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DeleteResult({'n': 0, 'electionId': ObjectId('7fffffff000000000000003a'), 'opTime': {'ts': Timestamp(1736447917, 1), 't': 58}, 'ok': 1.0, '$clusterTime': {'clusterTime': Timestamp(1736447917, 1), 'signature': {'hash': b'Y\\xd1\\x15\\xa32fD\\x0fx\\xaf\\xbbp\\x13\\x19\\xb1QR)~\\xf7', 'keyId': 7421923411288391683}}, 'operationTime': Timestamp(1736447917, 1)}, acknowledged=True)"
            ]
          },
          "execution_count": 38,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "working_memory.delete_many({})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "7IzJEpJh3m4R"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "\n",
        "class WorkingMemoryData(BaseModel):\n",
        "    content: str\n",
        "    site_title: str\n",
        "    site_url: str\n",
        "    added_at: datetime = Field(default_factory=datetime.utcnow)\n",
        "    embedding: List[float] = Field(\n",
        "        ..., description=\"The embedding vector for the news article\"\n",
        "    )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "sqfrj_Xfgumh"
      },
      "outputs": [],
      "source": [
        "def my_ranking_function(query, documents, top_n):\n",
        "    return documents"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "id": "gZ_rk14thJjI"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "\n",
        "def save_document(document: dict) -> Optional[WorkingMemoryData]:\n",
        "    \"\"\"\n",
        "    Processes a document and converts it to a WorkingMemoryData instance.\n",
        "\n",
        "    Args:\n",
        "      document: A dictionary containing the raw document data\n",
        "\n",
        "    Returns:\n",
        "      WorkingMemoryData instance if document meets criteria, None otherwise\n",
        "    \"\"\"\n",
        "    # First check the score threshold\n",
        "    if document[\"score\"] < 0.5:\n",
        "        return None\n",
        "\n",
        "    # Generate embeddings for the content\n",
        "    embedding_vector = get_embeddings([document[\"content\"]])[0]\n",
        "\n",
        "    try:\n",
        "        # Create a WorkingMemoryData instance with validation\n",
        "        processed_document = WorkingMemoryData(\n",
        "            content=document[\"content\"],\n",
        "            site_title=document[\"title\"],\n",
        "            site_url=document[\"url\"],\n",
        "            embedding=embedding_vector,\n",
        "            # added_at will be set automatically by default_factory\n",
        "        )\n",
        "\n",
        "        return processed_document.dict()\n",
        "\n",
        "    except ValueError as e:\n",
        "        # Handle any validation errors from Pydantic\n",
        "        print(f\"Error creating WorkingMemoryData: {e}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "id": "szPRxvIDGNyb"
      },
      "outputs": [],
      "source": [
        "from tavily import TavilyHybridClient\n",
        "\n",
        "# Documentation on the hybridrag client: https://docs.tavily.com/docs/python-sdk/tavily-hybrid-rag/getting-started\n",
        "hybrid_rag = TavilyHybridClient(\n",
        "    api_key=os.environ.get(\"TAVILY_API_KEY\"),\n",
        "    db_provider=\"mongodb\",\n",
        "    collection=working_memory,\n",
        "    index=\"vector_index\",\n",
        "    embedding_function=get_embeddings,\n",
        "    embeddings_field=\"embedding\",\n",
        "    content_field=\"description\",\n",
        "    ranking_function=my_ranking_function,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "id": "sEl8B1cvMXmc"
      },
      "outputs": [],
      "source": [
        "query = \"Get me some news on electric cars if possible\"\n",
        "internet_search_results = hybrid_rag.search(query)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CrP4qEEUg9M2",
        "outputId": "54f05ab4-5f08-4377-bdc7-d552eaa5516f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{'content': 'Photo Galleries\\n'\n",
            "             'Most Popular\\n'\n",
            "             'Motor Authority Newsletter\\n'\n",
            "             'Sign up to get the latest performance and luxury automotive '\n",
            "             'news, delivered to your inbox daily!\\n'\n",
            "             ' Electric Cars\\n'\n",
            "             'The AMG version of the EQE SUV doesn’t have the fire and fury of '\n",
            "             'other models from Mercedes’ performance arm.\\n'\n",
            "             ' Will the jump-started VW brand really bring out a new '\n",
            "             'Aristocrat, or is just protecting IP?\\n'\n",
            "             'VW is working on an electric GTI but it might not be '\n",
            "             'Golf-based.\\n'\n",
            "             ' The 1,234-hp Lucid Air Sapphire is the quickest car ever to '\n",
            "             'grace the MA Best Car To Buy competition.\\n'\n",
            "             ' The 964 RSR is a dream car for 911 fans of a certain age, and '\n",
            "             'Everrati is looking to capitalize with an electric tribute.\\n',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.7156829},\n",
            " {'content': 'Electric Cars news & latest pictures from Newsweek.com Newsweek '\n",
            "             'pulls back the curtain on what goes has gone into developing '\n",
            "             \"Rivian's electric vehicle charging network. U.S. Electric \"\n",
            "             'vehicles improved but still less reliable than gas models: '\n",
            "             'survey Donald Trump is reportedly planning to scrap a $7,500 '\n",
            "             'federal consumer tax credit for electric vehicles. Gavin Newsom '\n",
            "             'prepared to challenge Trump on electric vehicle tax credits '\n",
            "             'Gavin Newsom prepared to challenge Trump on electric vehicle tax '\n",
            "             'credits Newsom proposes California offer state tax rebates for '\n",
            "             'electric vehicle purchases should Donald Trump eliminate the '\n",
            "             'federal EV tax credit. Getting rid of the federal rebate could '\n",
            "             'devastate the electric vehicle industry, but Tesla CEO Elon Musk '\n",
            "             \"doesn't mind. U.S. awards $3 billion for EV battery production \"\n",
            "             'to counter China',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.6617704},\n",
            " {'content': 'Read the latest electric vehicle news, recent EV reviews and EV '\n",
            "             'buying advice at Cars.com.',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.6453216},\n",
            " {'content': 'And Hard\\n'\n",
            "             'The 2025 Honda CR-V e:FCEV Is A Hydrogen Plug-In Hybrid, For '\n",
            "             'Real\\n'\n",
            "             'EV News\\n'\n",
            "             'Filter by:\\n'\n",
            "             'Why Is Motorcycle Racing Afraid Of This Electric Bike?\\n'\n",
            "             'Chinese Cars Would Get 125% Price Increase Under New Senate '\n",
            "             'Bill\\n'\n",
            "             'Watch Tesla Cybertruck Owner Shoot Bullets At His Truck With '\n",
            "             'Submachine Gun, Shotgun\\n'\n",
            "             \"'Mind-Blowing' Tesla Roadster Final Form To Debut But Then The \"\n",
            "             'Story Got Weird\\n'\n",
            "             'What Trump Got Wrong About EVs During His Michigan Speech\\n'\n",
            "             \"The Polestar 3 Can't Come Soon Enough\\n\"\n",
            "             'Tesla Arson Suspect Caught On Camera: Two Model Ys Burnt\\n'\n",
            "             'FreeWire’s New Pro Series DC Fast Chargers Can Also Power Your '\n",
            "             'Business\\n'\n",
            "             'Will Americans Buy This Tiny, Cute Electric Car?\\n'\n",
            "             ' It May Work\\n'\n",
            "             \"Watch Ford's 1,400-HP SuperVan Blast Its Way To Several Lap \"\n",
            "             'Records At Bathurst\\n'\n",
            "             'Electrification Leads To All-Wheel-Drive Dominance\\n'\n",
            "             'BYD Brings Denza Brand To Europe With Striking D9 Minivan\\n'\n",
            "             '2024 U.S. Electric Cars Listed From Lowest To Highest Energy '\n",
            "             'Consumption\\n'\n",
            "             'This Dodge Ram Pickup Was Destroyed After Rear-Ending A Tesla '\n",
            "             'Cybertruck Search for:\\n'\n",
            "             'Armored Glass Repels Tesla Cybertruck Smash-And-Grab Attempt\\n'\n",
            "             'Toyota’s New Engine Can Suck Carbon Out Of The Air\\n'\n",
            "             \"Hyundai Confirms Its Georgia 'Metaplant' EV Factory Is Opening \"\n",
            "             'Early\\n'\n",
            "             '2024 U.S. Electric Cars Compared By Price Per Mile Of EPA Range\\n'\n",
            "             'This Guy Told Us He Bought All The Cakes, Buick Wildcat Concept '\n",
            "             \"Could Inspire 'Exceptional By Design' EVs\\n\"\n",
            "             'Hyundai Kills All N Gasoline Performance Cars In Europe\\n'\n",
            "             'China Plug-In Car Sales Almost Doubled In January 2024\\n'\n",
            "             'Car Buying Service\\n'\n",
            "             'Get upfront price offers on local inventory.\\n',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.6347929},\n",
            " {'content': 'Electric Cars news - Today’s latest updates - CBS News CBS News '\n",
            "             'Miami investigative reporter Jim Defede and CBS News Texas '\n",
            "             'investigative reporter Brian New break down how lawmakers and '\n",
            "             'residents in their states view climate change amid natural '\n",
            "             'disasters. #### U.S. News lists its best electric and hybrid '\n",
            "             \"vehicles for 2024 Foreign automakers dominate U.S. News' list of \"\n",
            "             'the best new EVs and hybrids, while Tesla is shut out. #### '\n",
            "             'Latest CBS News Videos #### California councilwoman on '\n",
            "             'evacuations L.A. City Councilmember Nithya Raman told CBS News '\n",
            "             'Los Angeles the latest updates on the Sunset Fire burning in the '\n",
            "             \"Hollywood Hills on Wednesday evening. CBS News Los Angeles' Joy \"\n",
            "             'Benedict reports that some firefighters ran out of water, but '\n",
            "             'got help from other departments.',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.63125396},\n",
            " {'content': 'Although Genesis and Hyundai plan to make some of their EVs in '\n",
            "             'the U.S., the biggest, most expensive electric SUV planned for '\n",
            "             'the lineup will be Korean-made, according to a report and plant '\n",
            "             'announcement.\\n'\n",
            "             ' The Audi E-Tron SUV—now the Q8 E-Tron—topped the list, with '\n",
            "             'data showing it retained the highest ratio of its range in '\n",
            "             'freezing temps.\\n'\n",
            "             ' The Lucid Gravity will help the startup automaker break into '\n",
            "             'the heart of the automotive market with a three-row crossover '\n",
            "             'SUV.\\n'\n",
            "             ' Tesla may have installed the wrong airbag for Model S and Model '\n",
            "             'X owners who opted to switch from the available steering yoke '\n",
            "             'back to the steering wheel, or vice versa.\\n'\n",
            "             ' The GM luxury brand confirmed the Optiq as the \"entry point for '\n",
            "             'Cadillac’s EV lineup in North America,\" sitting below the '\n",
            "             'Lyriq.\\n',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.586926},\n",
            " {'content': 'China Plug-In Car Sales Almost Doubled In January 2024\\n'\n",
            "             '2024 Volkswagen ID.4 Starts At $39,735, Pro Models Get More '\n",
            "             'Powerful\\n'\n",
            "             'Armored Glass Repels Tesla Cybertruck Smash-And-Grab Attempt\\n'\n",
            "             'The Apple Car Is Finally Dead, Shrouded In Mystery Until The '\n",
            "             'End: Report\\n'\n",
            "             'Toyota’s New Engine Can Suck Carbon Out Of The Air\\n'\n",
            "             \"Hyundai Confirms Its Georgia 'Metaplant' EV Factory Is Opening \"\n",
            "             'Early\\n'\n",
            "             '2024 U.S. Electric Cars Compared By Price Per Mile Of EPA Range\\n'\n",
            "             'One-Year-Old Kia Niro EVs Are Depreciating But Then The Story '\n",
            "             'Got Weird\\n'\n",
            "             'What Trump Got Wrong About EVs During His Michigan Speech\\n'\n",
            "             \"The Polestar 3 Can't Come Soon Enough\\n\"\n",
            "             'Tesla Arson Suspect Caught On Camera: Two Model Ys Burnt\\n'\n",
            "             'The 2025 Honda CR-V e:FCEV Is A Hydrogen Plug-In Hybrid, For '\n",
            "             'Real\\n'\n",
            "             'FreeWire’s New Pro Series DC Fast Chargers Can Also Power Your '\n",
            "             'Business\\n'\n",
            "             'Will Americans Buy This Tiny, Cute Electric Car?\\n'\n",
            "             \" Buick Wildcat Concept Could Inspire 'Exceptional By Design' \"\n",
            "             'EVs\\n'\n",
            "             'Hyundai Kills All N Gasoline Performance Cars In Europe\\n'\n",
            "             \"American Test Of $11,500 BYD Seagull: 'This Doesn't Come Across \"\n",
            "             \"Cheap'\\n\"\n",
            "             'Features\\n'\n",
            "             'What To Do If You’ve Just Rented An Electric Car\\n'\n",
            "             'Is A Used Mini Cooper SE The Perfect Second Car?\\n'\n",
            "             ' Until The End: Report\\n'\n",
            "             'Reviews\\n'\n",
            "             'The e:NY1 Shows Honda Isn’t Trying Hard Enough On EVs\\n'\n",
            "             \"The 2024 Honda Prologue Should Tide You Over Til' Dinner's \"\n",
            "             'Ready\\n'\n",
            "             'We Have A Tesla Cybertruck. And Hard\\n'\n",
            "             'Reviews\\n'\n",
            "             'The e:NY1 Shows Honda Isn’t Trying Hard Enough On EVs\\n'\n",
            "             \"The 2024 Honda Prologue Should Tide You Over Til' Dinner's \"\n",
            "             'Ready\\n'\n",
            "             'We Have A Tesla Cybertruck.',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.562874},\n",
            " {'content': 'The company’s Cooper SE previously held the distinction of '\n",
            "             'having the lowest range of any EV available in the U.S. While '\n",
            "             'the Aceman will exceed that car’s 114 miles, it’s not expected '\n",
            "             'to break 300 miles. A smaller electric SUV that will bring '\n",
            "             'Rivian design within reach of the average car buyer, the R2 is '\n",
            "             'said to have a range of at least 300 miles, regardless of '\n",
            "             'powertrain. You can also stay up to date on the latest EV news '\n",
            "             'on the\\xa0TrueCar Blog\\xa0or on our\\xa0Electric Vehicles Hub. '\n",
            "             'And when you’re ready to buy an electric vehicle, you can use\\xa0'\n",
            "             'TrueCar\\xa0to shop and get an up-front, personalized offer from '\n",
            "             'a Certified Dealer.',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.559411},\n",
            " {'content': 'Best electric cars arriving in 2025 - Car News | CarsGuide Sell '\n",
            "             'my car Sign up / Sign in Welcome back! Sign up / Sign in New to '\n",
            "             'Carsguide? Sign up Welcome back! Sign in Help buy + sell Buy Buy '\n",
            "             'a car New What car should I buy? Sell Sell my car reviews '\n",
            "             'Reviews All reviewsBrowse over 9,000 car reviews FamilyFamily '\n",
            "             'focused reviews and advice for everything family car related. '\n",
            "             \"Here's what to look out for and buy smart Buying guides Electric \"\n",
            "             \"news News Latest newsWhat's happening in the automotive world \"\n",
            "             'Motor showsThe stars of the latest big events TechnologyThe '\n",
            "             \"latest and future car tech from around the world All adviceWe're \"\n",
            "             'here to help you with any car issues',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.5059329},\n",
            " {'content': \"Uncertainty over Trump's electric vehicle policies clouds 2025 \"\n",
            "             'forecast for carmakers | AP News AP News Alerts Keep your pulse '\n",
            "             'on the news with breaking news alerts from The AP.The Morning '\n",
            "             'Wire Our flagship newsletter breaks down the biggest headlines '\n",
            "             'of the day.Ground Game Exclusive insights and key stories from '\n",
            "             'the world of politics.Beyond the Story Executive Editor Julie '\n",
            "             'Pace brings you behind the scenes of the AP newsroom.AP Top 25 '\n",
            "             'Poll Alerts Get email alerts for every college football Top 25 '\n",
            "             \"Poll release.AP Top 25 Women's Basketball Poll Alerts Women's \"\n",
            "             'college basketball poll alerts and updates. NEW YORK (AP) — '\n",
            "             'Electric vehicle demand is expected to keep rising this year, '\n",
            "             'but uncertainty over policy changes and tariffs is clouding the '\n",
            "             'forecast.',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.36466494}]\n"
          ]
        }
      ],
      "source": [
        "pprint.pprint(internet_search_results)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KczNnDMRIrGy",
        "outputId": "20712b28-c6b5-4865-dc16-7022350646d8"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "<ipython-input-41-18507b6d9b9a>:30: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.10/migration/\n",
            "  return processed_document.dict()\n"
          ]
        }
      ],
      "source": [
        "query = \"Get me some news on electric cars if possible\"\n",
        "max_foreign = 5\n",
        "max_local = 5\n",
        "\n",
        "internet_search_results = hybrid_rag.search(\n",
        "    query, max_local=max_local, max_foreign=max_foreign, save_foreign=save_document\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F4rB0KLII__i",
        "outputId": "4e2d7000-077f-40ac-c48f-059f97df79e6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{'content': 'Photo Galleries\\n'\n",
            "             'Most Popular\\n'\n",
            "             'Motor Authority Newsletter\\n'\n",
            "             'Sign up to get the latest performance and luxury automotive '\n",
            "             'news, delivered to your inbox daily!\\n'\n",
            "             ' Electric Cars\\n'\n",
            "             'The AMG version of the EQE SUV doesn’t have the fire and fury of '\n",
            "             'other models from Mercedes’ performance arm.\\n'\n",
            "             ' Will the jump-started VW brand really bring out a new '\n",
            "             'Aristocrat, or is just protecting IP?\\n'\n",
            "             'VW is working on an electric GTI but it might not be '\n",
            "             'Golf-based.\\n'\n",
            "             ' The 1,234-hp Lucid Air Sapphire is the quickest car ever to '\n",
            "             'grace the MA Best Car To Buy competition.\\n'\n",
            "             ' The 964 RSR is a dream car for 911 fans of a certain age, and '\n",
            "             'Everrati is looking to capitalize with an electric tribute.\\n',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.7156829},\n",
            " {'content': 'Electric Cars news & latest pictures from Newsweek.com Newsweek '\n",
            "             'pulls back the curtain on what goes has gone into developing '\n",
            "             \"Rivian's electric vehicle charging network. U.S. Electric \"\n",
            "             'vehicles improved but still less reliable than gas models: '\n",
            "             'survey Donald Trump is reportedly planning to scrap a $7,500 '\n",
            "             'federal consumer tax credit for electric vehicles. Gavin Newsom '\n",
            "             'prepared to challenge Trump on electric vehicle tax credits '\n",
            "             'Gavin Newsom prepared to challenge Trump on electric vehicle tax '\n",
            "             'credits Newsom proposes California offer state tax rebates for '\n",
            "             'electric vehicle purchases should Donald Trump eliminate the '\n",
            "             'federal EV tax credit. Getting rid of the federal rebate could '\n",
            "             'devastate the electric vehicle industry, but Tesla CEO Elon Musk '\n",
            "             \"doesn't mind. U.S. awards $3 billion for EV battery production \"\n",
            "             'to counter China',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.6617704},\n",
            " {'content': 'Read the latest electric vehicle news, recent EV reviews and EV '\n",
            "             'buying advice at Cars.com.',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.6453216},\n",
            " {'content': 'Electric Cars news - Today’s latest updates - CBS News CBS News '\n",
            "             'Miami investigative reporter Jim Defede and CBS News Texas '\n",
            "             'investigative reporter Brian New break down how lawmakers and '\n",
            "             'residents in their states view climate change amid natural '\n",
            "             'disasters. #### U.S. News lists its best electric and hybrid '\n",
            "             \"vehicles for 2024 Foreign automakers dominate U.S. News' list of \"\n",
            "             'the best new EVs and hybrids, while Tesla is shut out. #### '\n",
            "             'Latest CBS News Videos #### California councilwoman on '\n",
            "             'evacuations L.A. City Councilmember Nithya Raman told CBS News '\n",
            "             'Los Angeles the latest updates on the Sunset Fire burning in the '\n",
            "             \"Hollywood Hills on Wednesday evening. CBS News Los Angeles' Joy \"\n",
            "             'Benedict reports that some firefighters ran out of water, but '\n",
            "             'got help from other departments.',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.63125396},\n",
            " {'content': 'China Plug-In Car Sales Almost Doubled In January 2024\\n'\n",
            "             '2024 Volkswagen ID.4 Starts At $39,735, Pro Models Get More '\n",
            "             'Powerful\\n'\n",
            "             'Armored Glass Repels Tesla Cybertruck Smash-And-Grab Attempt\\n'\n",
            "             'The Apple Car Is Finally Dead, Shrouded In Mystery Until The '\n",
            "             'End: Report\\n'\n",
            "             'Toyota’s New Engine Can Suck Carbon Out Of The Air\\n'\n",
            "             \"Hyundai Confirms Its Georgia 'Metaplant' EV Factory Is Opening \"\n",
            "             'Early\\n'\n",
            "             '2024 U.S. Electric Cars Compared By Price Per Mile Of EPA Range\\n'\n",
            "             'One-Year-Old Kia Niro EVs Are Depreciating But Then The Story '\n",
            "             'Got Weird\\n'\n",
            "             'What Trump Got Wrong About EVs During His Michigan Speech\\n'\n",
            "             \"The Polestar 3 Can't Come Soon Enough\\n\"\n",
            "             'Tesla Arson Suspect Caught On Camera: Two Model Ys Burnt\\n'\n",
            "             'The 2025 Honda CR-V e:FCEV Is A Hydrogen Plug-In Hybrid, For '\n",
            "             'Real\\n'\n",
            "             'FreeWire’s New Pro Series DC Fast Chargers Can Also Power Your '\n",
            "             'Business\\n'\n",
            "             'Will Americans Buy This Tiny, Cute Electric Car?\\n'\n",
            "             \" Buick Wildcat Concept Could Inspire 'Exceptional By Design' \"\n",
            "             'EVs\\n'\n",
            "             'Hyundai Kills All N Gasoline Performance Cars In Europe\\n'\n",
            "             \"American Test Of $11,500 BYD Seagull: 'This Doesn't Come Across \"\n",
            "             \"Cheap'\\n\"\n",
            "             'Features\\n'\n",
            "             'What To Do If You’ve Just Rented An Electric Car\\n'\n",
            "             'Is A Used Mini Cooper SE The Perfect Second Car?\\n'\n",
            "             ' Until The End: Report\\n'\n",
            "             'Reviews\\n'\n",
            "             'The e:NY1 Shows Honda Isn’t Trying Hard Enough On EVs\\n'\n",
            "             \"The 2024 Honda Prologue Should Tide You Over Til' Dinner's \"\n",
            "             'Ready\\n'\n",
            "             'We Have A Tesla Cybertruck. And Hard\\n'\n",
            "             'Reviews\\n'\n",
            "             'The e:NY1 Shows Honda Isn’t Trying Hard Enough On EVs\\n'\n",
            "             \"The 2024 Honda Prologue Should Tide You Over Til' Dinner's \"\n",
            "             'Ready\\n'\n",
            "             'We Have A Tesla Cybertruck.',\n",
            "  'origin': 'foreign',\n",
            "  'score': 0.56258565}]\n"
          ]
        }
      ],
      "source": [
        "pprint.pprint(internet_search_results)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {
        "id": "OOueWe1AhnpS"
      },
      "outputs": [],
      "source": [
        "@agent.tool_plain\n",
        "def get_search_results_from_internet_search(user_query):\n",
        "    \"\"\"Use Tavily to get search results from the internet.\"\"\"\n",
        "    return hybrid_rag.search(\n",
        "        user_query,\n",
        "        max_local=max_local,\n",
        "        max_foreign=max_foreign,\n",
        "        save_foreign=save_document,\n",
        "    )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 138,
      "metadata": {
        "id": "t9EOO8nJiyrf"
      },
      "outputs": [],
      "source": [
        "# from tavily import TavilyClient\n",
        "\n",
        "# tavily_client = TavilyClient(api_key=os.environ.get(\"TAVILY_API_KEY\"))\n",
        "\n",
        "# @agent.tool_plain\n",
        "# def extract_data_from_urls(urls):\n",
        "#   \"\"\" Extract content from a urls in a given list \"\"\"\n",
        "#   return tavily_client.extract(urls=urls, include_images=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CgEU1MGxj8uq",
        "outputId": "e5df1309-5921-462f-b142-b26c8cead28b"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "<ipython-input-41-18507b6d9b9a>:30: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.10/migration/\n",
            "  return processed_document.dict()\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Here is some recent news on electric cars:\n",
            "\n",
            "1. **SK Signet Inks Deal with Francis Energy for Ultra-Fast EV Chargers:**\n",
            "   - **Date:** July 18, 2023\n",
            "   - **Details:** SK Signet signed a deal with Francis Energy to supply more than 1000 ultra-fast EV chargers in the US. Francis Energy is currently the fourth-largest fast charger operator in the United States.\n",
            "   - [Read more](https://www.econotimes.com/SK-signet-Inks-Deal-with-Francis-Energy-for-the-Supply-of-Ultra-Fast-EV-Chargers-to-the-US-1659601)\n",
            "\n",
            "2. **YS Tech Working Closely with Chinese Car Vendors:**\n",
            "   - **Date:** March 10, 2023\n",
            "   - **Details:** Automotive cooling fan supplier YS Tech is collaborating with Chinese customers and is anticipating a new Chinese government policy to boost its EV sector.\n",
            "   - [Read more](https://www.digitimes.com/news/a20230309PD211/automotive-china-ev+green-energy-ys-tech.html)\n",
            "\n",
            "3. **MotorTrend's New Electric Car Models:**\n",
            "   - 2025 Dodge Charger Sixpack is reportedly fast-tracked.\n",
            "   - New models like the 2025 Porsche Taycan and 2026 Cadillac Vistiq are highlighted, promising more excellence and sharp design in electric vehicle offerings.\n",
            "   - Mercedes-AMG is targeting Porsche with a high-performance electric SUV.\n",
            "\n",
            "4. **Hyundai's Electric Models and Plans:**\n",
            "   - Although Genesis and Hyundai plan to manufacture some EVs in the U.S., their largest electric SUV will continue to be made in Korea.\n",
            "   - Hyundai confirmed its \"Metaplant\" EV factory in Georgia will open earlier than planned.\n",
            "\n",
            "5. **General Observations:**\n",
            "   - The Audi Q8 E-Tron retains a high range in freezing temperatures.\n",
            "   - Tesla faced issues with incorrectly installed airbags for models switching between steering wheel configurations.\n",
            "\n",
            "For more in-depth reviews and buying advice on electric vehicles, you can visit platforms like Cars.com.\n"
          ]
        }
      ],
      "source": [
        "results = agent.run_sync(\n",
        "    \"Get me some news on electric cars if possible\", deps=MongoDeps\n",
        ")\n",
        "print(results.data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rzSi6LLflzb6",
        "outputId": "3193cda3-0204-47eb-8401-29e754f119ff"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[ModelRequest(parts=[SystemPromptPart(content='You get the latest news based on a user query', dynamic_ref=None, part_kind='system-prompt'), UserPromptPart(content='Get me some news on electric cars if possible', timestamp=datetime.datetime(2025, 1, 9, 18, 39, 45, 435371, tzinfo=datetime.timezone.utc), part_kind='user-prompt')], kind='request'),\n",
              " ModelResponse(parts=[ToolCallPart(tool_name='retrieve_information_from_knowledge_base', args=ArgsJson(args_json='{\"user_query\": \"electric cars news\"}'), tool_call_id='call_c10S8UWOpb2Pxn8U56W4sLvm', part_kind='tool-call'), ToolCallPart(tool_name='get_search_results_from_internet_search', args=ArgsJson(args_json='{\"user_query\": \"electric cars news\"}'), tool_call_id='call_xcF2cH3F4K9glpRLCAJo0Q44', part_kind='tool-call')], timestamp=datetime.datetime(2025, 1, 9, 18, 39, 45, tzinfo=datetime.timezone.utc), kind='response'),\n",
              " ModelRequest(parts=[ToolReturnPart(tool_name='retrieve_information_from_knowledge_base', content='[{\\'companyName\\': \\'01Synergy\\', \\'companyUrl\\': \\'https://hackernoon.com/company/01synergy\\', \\'published_at\\': \\'2023-07-18 08:31:00\\', \\'title\\': \\'SK signet Inks Deal with Francis Energy for the Supply of Ultra-Fast EV Chargers to the US\\', \\'description\\': \\'SK Signet revealed it signed a deal with Francis Energy for an order of more than 1000 EV chargers. The latter is currently the fourth-largest fast charger operator in the United States and it has agreed to a\\', \\'url\\': \\'https://www.econotimes.com/SK-signet-Inks-Deal-with-Francis-Energy-for-the-Supply-of-Ultra-Fast-EV-Chargers-to-the-US-1659601\\', \\'score\\': 0.7703076601028442}, {\\'companyName\\': \\'01Synergy\\', \\'companyUrl\\': \\'https://hackernoon.com/company/01synergy\\', \\'published_at\\': \\'2023-03-10 02:28:00\\', \\'title\\': \\'YS Tech working closely with China car vendors\\', \\'description\\': \"Automotive cooling fan supplier Yen Sun Technology (YS Tech) said it will work closely with Chinese customers and is anticipating a new Chinese government policy to boost the country\\'s EV sector.\", \\'url\\': \\'https://www.digitimes.com/news/a20230309PD211/automotive-china-ev+green-energy-ys-tech.html\\', \\'score\\': 0.7642202377319336}, {\\'companyName\\': \\'01Synergy\\', \\'companyUrl\\': \\'https://hackernoon.com/company/01synergy\\', \\'published_at\\': \\'2023-03-10 02:28:00\\', \\'title\\': \\'YS Tech working closely with China car vendors\\', \\'description\\': \"Automotive cooling fan supplier Yen Sun Technology (YS Tech) said it will work closely with Chinese customers and is anticipating a new Chinese government policy to boost the country\\'s EV sector.\", \\'url\\': \\'https://www.digitimes.com/news/a20230309PD211/automotive-china-ev+green-energy-ys-tech.html\\', \\'score\\': 0.7642202377319336}, {\\'companyName\\': \\'01Synergy\\', \\'companyUrl\\': \\'https://hackernoon.com/company/01synergy\\', \\'published_at\\': \\'2023-03-10 02:28:00\\', \\'title\\': \\'YS Tech working closely with China car vendors\\', \\'description\\': \"Automotive cooling fan supplier Yen Sun Technology (YS Tech) said it will work closely with Chinese customers and is anticipating a new Chinese government policy to boost the country\\'s EV sector.\", \\'url\\': \\'https://www.digitimes.com/news/a20230309PD211/automotive-china-ev+green-energy-ys-tech.html?chid=13\\', \\'score\\': 0.7642202377319336}, {\\'companyName\\': \\'10Clouds\\', \\'companyUrl\\': \\'https://hackernoon.com/company/10clouds\\', \\'published_at\\': \\'2023-01-30 14:08:00\\', \\'title\\': \\'Investing in Cleaner Technology: Lesser-Known Areas of Innovation to Watch\\', \\'description\\': \\'That said January has seen two powerful news events that may have slipped under your radar but that have the potential to have enormous impact on the efforts towards cleaner energy. Here we are going to look at cleaner energy investment opportunities that can help bridge the gap between where the science and our needs are today versus where we want to be in the future.\\', \\'url\\': \\'https://www.nasdaq.com/articles/investing-in-cleaner-technology%3A-lesser-known-areas-of-innovation-to-watch\\', \\'score\\': 0.7604036331176758}]', tool_call_id='call_c10S8UWOpb2Pxn8U56W4sLvm', timestamp=datetime.datetime(2025, 1, 9, 18, 39, 47, 477184, tzinfo=datetime.timezone.utc), part_kind='tool-return'), ToolReturnPart(tool_name='get_search_results_from_internet_search', content=[{'score': 0.8463680744171143, 'origin': 'local'}, {'score': 0.8341740965843201, 'origin': 'local'}, {'score': 0.8332723379135132, 'origin': 'local'}, {'score': 0.8189547657966614, 'origin': 'local'}, {'score': 0.8049435615539551, 'origin': 'local'}, {'content': 'MotorTrend | News 2025 Dodge Charger Sixpack Gas Muscle Car is Reportedly Getting Fast-Tracked ---------------------------------------------------------------------------- Andrew Beckford | Nov 12, 2024 MotorTrend | First Look 2025 Porsche Taycan First Look: More Excellence ----------------------------------------------- Frank Markus | Nov 12, 2024 MotorTrend | First Look 2026 Cadillac Vistiq First Look: Sharp All-Electric 3-Row Family SUV -------------------------------------------------------------------- Alex Leanse | Nov 12, 2024 MotorTrend | News Next-Gen Chevrolet Bolt EV Kills Off a Cadillac SUV --------------------------------------------------- Justin Westbrook | Nov 11, 2024 MotorTrend | Future Cars Mercedes-AMG Going After Porsche With High-Performance Electric SUV ------------------------------------------------------------------- Justin Westbrook | Nov 7, 2024', 'score': 0.82082915, 'origin': 'foreign'}, {'content': 'Photo Galleries\\nMost Popular\\nMotor Authority Newsletter\\nSign up to get the latest performance and luxury automotive news, delivered to your inbox daily!\\n Electric Cars\\nThe AMG version of the EQE SUV doesn’t have the fire and fury of other models from Mercedes’ performance arm.\\n Will the jump-started VW brand really bring out a new Aristocrat, or is just protecting IP?\\nVW is working on an electric GTI but it might not be Golf-based.\\n The 1,234-hp Lucid Air Sapphire is the quickest car ever to grace the MA Best Car To Buy competition.\\n The 964 RSR is a dream car for 911 fans of a certain age, and Everrati is looking to capitalize with an electric tribute.\\n', 'score': 0.81665546, 'origin': 'foreign'}, {'content': 'Although Genesis and Hyundai plan to make some of their EVs in the U.S., the biggest, most expensive electric SUV planned for the lineup will be Korean-made, according to a report and plant announcement.\\n The Audi E-Tron SUV—now the Q8 E-Tron—topped the list, with data showing it retained the highest ratio of its range in freezing temps.\\n The Lucid Gravity will help the startup automaker break into the heart of the automotive market with a three-row crossover SUV.\\n Tesla may have installed the wrong airbag for Model S and Model X owners who opted to switch from the available steering yoke back to the steering wheel, or vice versa.\\n The GM luxury brand confirmed the Optiq as the \"entry point for Cadillac’s EV lineup in North America,\" sitting below the Lyriq.\\n', 'score': 0.769306, 'origin': 'foreign'}, {'content': \"And Hard\\nThe 2025 Honda CR-V e:FCEV Is A Hydrogen Plug-In Hybrid, For Real\\nEV News\\nFilter by:\\nWhy Is Motorcycle Racing Afraid Of This Electric Bike?\\nChinese Cars Would Get 125% Price Increase Under New Senate Bill\\nWatch Tesla Cybertruck Owner Shoot Bullets At His Truck With Submachine Gun, Shotgun\\n'Mind-Blowing' Tesla Roadster Final Form To Debut But Then The Story Got Weird\\nWhat Trump Got Wrong About EVs During His Michigan Speech\\nThe Polestar 3 Can't Come Soon Enough\\nTesla Arson Suspect Caught On Camera: Two Model Ys Burnt\\nFreeWire’s New Pro Series DC Fast Chargers Can Also Power Your Business\\nWill Americans Buy This Tiny, Cute Electric Car?\\n It May Work\\nWatch Ford's 1,400-HP SuperVan Blast Its Way To Several Lap Records At Bathurst\\nElectrification Leads To All-Wheel-Drive Dominance\\nBYD Brings Denza Brand To Europe With Striking D9 Minivan\\n2024 U.S. Electric Cars Listed From Lowest To Highest Energy Consumption\\nThis Dodge Ram Pickup Was Destroyed After Rear-Ending A Tesla Cybertruck Search for:\\nArmored Glass Repels Tesla Cybertruck Smash-And-Grab Attempt\\nToyota’s New Engine Can Suck Carbon Out Of The Air\\nHyundai Confirms Its Georgia 'Metaplant' EV Factory Is Opening Early\\n2024 U.S. Electric Cars Compared By Price Per Mile Of EPA Range\\nThis Guy Told Us He Bought All The Cakes, Buick Wildcat Concept Could Inspire 'Exceptional By Design' EVs\\nHyundai Kills All N Gasoline Performance Cars In Europe\\nChina Plug-In Car Sales Almost Doubled In January 2024\\nCar Buying Service\\nGet upfront price offers on local inventory.\\n\", 'score': 0.76826453, 'origin': 'foreign'}, {'content': 'Read the latest electric vehicle news, recent EV reviews and EV buying advice at Cars.com.', 'score': 0.7646988, 'origin': 'foreign'}], tool_call_id='call_xcF2cH3F4K9glpRLCAJo0Q44', timestamp=datetime.datetime(2025, 1, 9, 18, 39, 50, 864331, tzinfo=datetime.timezone.utc), part_kind='tool-return')], kind='request'),\n",
              " ModelResponse(parts=[TextPart(content='Here is some recent news on electric cars:\\n\\n1. **SK Signet Inks Deal with Francis Energy for Ultra-Fast EV Chargers:**\\n   - **Date:** July 18, 2023\\n   - **Details:** SK Signet signed a deal with Francis Energy to supply more than 1000 ultra-fast EV chargers in the US. Francis Energy is currently the fourth-largest fast charger operator in the United States.\\n   - [Read more](https://www.econotimes.com/SK-signet-Inks-Deal-with-Francis-Energy-for-the-Supply-of-Ultra-Fast-EV-Chargers-to-the-US-1659601)\\n\\n2. **YS Tech Working Closely with Chinese Car Vendors:**\\n   - **Date:** March 10, 2023\\n   - **Details:** Automotive cooling fan supplier YS Tech is collaborating with Chinese customers and is anticipating a new Chinese government policy to boost its EV sector.\\n   - [Read more](https://www.digitimes.com/news/a20230309PD211/automotive-china-ev+green-energy-ys-tech.html)\\n\\n3. **MotorTrend\\'s New Electric Car Models:**\\n   - 2025 Dodge Charger Sixpack is reportedly fast-tracked.\\n   - New models like the 2025 Porsche Taycan and 2026 Cadillac Vistiq are highlighted, promising more excellence and sharp design in electric vehicle offerings.\\n   - Mercedes-AMG is targeting Porsche with a high-performance electric SUV.\\n\\n4. **Hyundai\\'s Electric Models and Plans:**\\n   - Although Genesis and Hyundai plan to manufacture some EVs in the U.S., their largest electric SUV will continue to be made in Korea.\\n   - Hyundai confirmed its \"Metaplant\" EV factory in Georgia will open earlier than planned.\\n\\n5. **General Observations:**\\n   - The Audi Q8 E-Tron retains a high range in freezing temperatures.\\n   - Tesla faced issues with incorrectly installed airbags for models switching between steering wheel configurations.\\n\\nFor more in-depth reviews and buying advice on electric vehicles, you can visit platforms like Cars.com.', part_kind='text')], timestamp=datetime.datetime(2025, 1, 9, 18, 39, 50, tzinfo=datetime.timezone.utc), kind='response')]"
            ]
          },
          "execution_count": 49,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "results.all_messages()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "00a23ef470614479aa83f44599c78c75": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1ed1495a5d184329a2ed7a337f166f0f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "20ac8b6e85984a7caa9fa25ecee2fefd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f6cb7ce3936544bfa36493ba1e52df54",
            "max": 42,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1ed1495a5d184329a2ed7a337f166f0f",
            "value": 42
          }
        },
        "2228e72308b04108ba71d63834614646": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "32810c0d52ce427aa2b7ac56c7c773b3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_df49ab197ef646ab8ef0054c3f7356d4",
              "IPY_MODEL_20ac8b6e85984a7caa9fa25ecee2fefd",
              "IPY_MODEL_54c36c1e4b124d1ea8a1761ec50bad89"
            ],
            "layout": "IPY_MODEL_f690099db2294a1a8bfe6844fc164c69"
          }
        },
        "54c36c1e4b124d1ea8a1761ec50bad89": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ab5f3f777ba246ce9b044e2658109bb3",
            "placeholder": "​",
            "style": "IPY_MODEL_00a23ef470614479aa83f44599c78c75",
            "value": " 42/42 [00:00&lt;00:00, 19.97it/s]"
          }
        },
        "90c496cef99d44a2a9158f1ee6582873": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ab5f3f777ba246ce9b044e2658109bb3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "df49ab197ef646ab8ef0054c3f7356d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_90c496cef99d44a2a9158f1ee6582873",
            "placeholder": "​",
            "style": "IPY_MODEL_2228e72308b04108ba71d63834614646",
            "value": "Resolving data files: 100%"
          }
        },
        "f690099db2294a1a8bfe6844fc164c69": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f6cb7ce3936544bfa36493ba1e52df54": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "state": {}
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
